nvidia/jetson-orin-nano-super: Enablement for GPU acceleration#23
Conversation
2bef2a4 to
42e48b2
Compare
d887b4c to
b8178d3
Compare
|
This last push tries to be less clever about how the drivers are being loaded, which it turns out is needed, since the reason it was working (somehow always) on my system was likely due to a race condition. When loading the same system configuration from the NVMe drive compared to a "usb installer iso" build, the latter wouldn't work, and I'd be reproducibly hitting the same issue my colleagues were. |
55d4959 to
7335aed
Compare
See the next commit.
This will be used to make the style check only apply to the locked Nixpkgs version.
It's redundant because at the `flake.nix` level this was already being handled. It's wrong because it's using the `lib` from the `flake-parts` input, which ***is not*** the `lib` from the `nixpkgs` or `pkgs` input! So whatever override we were passing for `nixpkgs` wouldn't affect that `lib`, and thus the `version` on the `lib` was whatever the `flake-parts`'s Nixpkgs was at.
This is one way Flakes let us down: there is no way to make the `check` run only when inputs were not overridden, or additional ways to affect the evaluation. So, just like Sisyphus, we need to push the version number up every time the formatting rules change in the formatter that is on the `nixos-unstable` release we track, compared to whichever versions we run as additional CI checks. Instead of handling this manually, let's just look at the locked version number from the release channel URL.
This unshackles the evaluation of the package from the NixOS evaluation.
Turns out we already basically were at r36.5, and this will have been the cause of some of the issues getting the GPU drivers going.
This is *basically* the open GPU driver codebase, but in a special Tegra configuration. It also is a different branch, with a version number that may not be matching anything equivalent on the open GPU drivers. This is not sufficient to have hardware acceleration. The proprietary drivers also have to be used. The proprietary driver version *must* match the version of the this display driver, or else things may not work, or worse.
…nel-display-driver This was backported as it looked like it may be able to fix the GPU driver usage with mainline 6.12. It did not. This did fix the `fbdev` parameter for 6.12.
This is not forced on end-users. This is provided because it *may* be needed to better support all features of the platform. Even with OOT drivers, some features won't work with mainline, or worse.
The `nvidia-l4t` package includes the proprietary libraries, the ones we don't have the equivalent sources to. These libraries are where the GPU acceleration lives.
98ac555 to
578f17e
Compare
This is the modern convention on Linux systems. Access to device nodes is mediated through ACLs being given by the login manager (not the display manager). This ensures common NixOS module configuration *just works* without needing to change the groups of greeters or compositors to include `video`. This fixes "just" using gdm and gnome (when `card0` is `nvidia-drm`). Also fix the `mknod` path at the same time.
Also introduce `enableStage1KernelModules` to allow end-users the option to disable the modules. This *should* generally be done for options that are lists, as those are poorly composable, and removing an element from a list in the modules system is generally hard.
When users otherwise don't explicitly care, use 6.12, as `nvidia-oot` only claims to support up to 6.15, and 6.12 is the prior LTS. This only sets the kernel using `mkDefault`, so this will not affect any selection users have made for themselves.
This should end-up being accurate when the modules needing it are enabled, but just in case, make it so `/run/opengl-driver` will be automatically enabled when hardware acceleration is enabled.
This is used to solve the issue where the NVIDIA drivers are (still) unable to take control of the device when they are used by a DRM driver. What's funny here is that this makes the graphics driver work on mainline 6.12, whereas when using `FBDEV` in the kernel config, starting xorg would reset the board.
578f17e to
9e114c5
Compare
…uffer and enabling hardware acceleration This should help end-users who might be disabling that quirk to figure out issues, and leaving it accidentally enabled.
…ules... This will be managed in the follow-up commit. Refer to it for the rationale
…ever way The clever way (trying to “react” on system-wide changes) ended-up having a race condition that could be triggered in unclear circumstances... and that race condition *was apparently why it worked*. So this rework instead implements that race condition as a feature, instead of relying on my good luck. This is still implemented as a `udev` rule as it is the easiest and most robust method to implement the required dance here: - Load `tegra-drm` such that it takes `card1` - Unbind `simpledrm` such that it frees `card0` - Load `nvidia-drm` such that it takes `card0` This, it turns out, is the secret sauce to make wayland compositors *just work* with the L4T drivers. The load order of the modules was already known to matter a lot: the `nvidia-drm` driver *must* be loaded after `tegra-drm`. And yes, the `tegra-drm` module *needs* to be loaded for it to work. The real difference compared to before is that having `card0` taken by the `nvidia-drm` driver means that, without some special proprietary sauce like in the X11 L4T driver or manual configuration, the software using DRM *should* pick the appropriate card by default. This matters when some compositors don't really expose a good way to pick a given `card` or `render` node. In an ideal world, either the `tegra-drm` would have its devices unregistered when `nvidia-drm` is loaded, or we could affect the allocation of the `minor_str` of the DRM devices so that a given device is prefered, or `nvidia-drm` could be loaded first without breaking. Alas, we're beholden to the drivers we are given. This strategy *does not* work when `simpledrm` is not in use. As of this commit, the use of the vendor kernel as-configured without `simpledrm` wouldn't work as there is nothing tying the `tegra-drm`+`nvidia-drm` driver loading to the classic `fbdev` driver. Such systems were *anyway* broken in the same way previously.
While this is not supported to *build*, we have CI checks against 24.05, so let's make CI work.
Let's not rely on `fbdev`, even though *it nominally works* for the vendor kernel. Both the vendor kernel and mainline should nominally work like the other for the system integration. And that means that it should enable the `tegra-drm` and `nvidia-drm` modules with the same semantics. When using the `fbdev` setup, the `tegra-drm` and `nvidia-drm` *are* effectively loaded (despite the blacklist?) but are loaded in a way where `card0` is the `tegra-drm` device, which would be a notable difference for non-X11 graphical use-cases.
…module Turns out the stack is ***very*** fussy, and there is *something* that can happen while loading the drivers, if it happens while the system is kinda busy, and in concurrence with a lot of other things. The reason this is assumed to be some race condition or similar situation is that the more things are made to go slowly, for example with sleep or by adding dynamic debug prints to the kernel, the more likely things are to work. So let's stop being clever and reactive about the environment. Instead, let's just brute-force our way to a working display output, and make the whole world wait for the driver to load. But only when the graphical target is involved. I believe this is something that the vendor is having difficulties with, as other distributions require turning of “SOC Display Hand-off” in the platform firmware for accelerated display support, and their newer Jetpack platform, for AGX Thor, also requires it for the vendor system. The fact that the vendor ships their kernel with fbdev also points toward that being true.
9e114c5 to
1e011a5
Compare
|
The recent changes don't fix the doom loop of fabric errors for me. For posterity:
jetson-orin-nano-with-dp-dell-display.log |
blitz
left a comment
There was a problem hiding this comment.
We have to leave debugging this for another day. From my end this can be merged.
1988b46
|
While doing final checks for reproducing the issue @blitz was facing, I discovered that on my local testing setup, using the DisplayPort port on a display reproduced the failure, while using the HDMI port on the same display did not. All my prior testing was done on an HDMI USB capture card. So if enabling the driver fails to enable the display, and keeps the device basically locked-up via continuous prints on the (serial) console, try an HDMI input. YMMV. |
What works:
glxandeglsupportWhat needs more investigative work:
gbmplatform undereglinfofailswaylandplatform undereglinfo) end-up usingllvmipesoftware renderingThese screnshots were taken using a capture card, from an X11 session:
This is YMMV
All of the pain setting up the display driver is likely caused by their driver stack being barely able to work in their very narrow supported use-case. The moment drivers load in a different order, or are configured differently, all bets are off about whether or not it's going to work.
There are two big issues here:
card0usageInability to take over the display peripherals
(Open for the details)
When the platform firmware leaves a
simple-framebuffercompatible, the driver stack is either unable to take over (withsimpledrm) or relies on older specific behaviours related to legacyfbdevdevices. Anefifbshould also break in the same manner.simple-framebuffersetup.simple-framebufferand disabling GPU acceleration.*driver*option in file*/etc/X11/xorg.conf*from*nvidia*to*FBDev*”nvidia-drmtaking overfbdevand specificallyfbdev.efifborsimple-framebufferwhen paired with the vendor drivers:simpledrm, and that this is the workaround for that.This is also upheld by the fact that manually doing the work (e.g. unbinding the
simpledrmdriver, as we do here) can make the driver work.Problematic setup for
card0usage(Open for the details)
This is less of a direct problem, and more of a consequence of their setup.
This is not an issue for X11 as the proprietary X11 driver seems to implement bespoke logic to pick the correct card when
tegra-drmis loaded.Wayland compositors and other DRM users like to use
/dev/dri/card0as the default, or sometimes relatively hardcoded card to use.Since the
nvidia-drmdriver (apparently) needs to havetegra-drmloaded first, and sincetegra-drmhas its ownminorassigned, which will be the first available, thetegra-drmdriver will have thecard0device, whilenvidia-drmwill have thecard1identifier.This is worked around when
simpledrmis present in our systems by makingtegra-drmload whilecard0is active. This means thattegra-drmusescard1, and whensimpledrmis dropped thecard0device is freed fornvidia-drm.This could be improved by forcing a DRM driver built for the purpose of taking
card0to be loaded, instead of relying onsimpledrmto be present.This is not strictly an issue with
nvidia-drm, but an issue with the broader DRM-using ecosystem, where there is no common way to point to a specific device to be used when it's notcard0. Thus, there is no proper way to point to/dev/dri/by-pathin a generic manner, if pointing to a path is even supported.